Two step gaze interaction

ABSTRACT

A computer system can be controlled with non-contact inputs, such as eye-tracking devices. A computer can enlarge a portion of a display adjacent a first gaze target in response to detecting a first action (e.g., pressing a touchpad). The computer can then allow a user to position a second gaze target in the enlarged portion (e.g., by looking at the desired location) and perform a second action in order to perform a computer function at that location. The enlarging can allow a user to identify a desired location for a computer function (e.g., selecting an icon) with greater precision.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 14/195,743 filed Mar. 3, 2014 which claims the benefit of U.S.Provisional Patent Application No. 61/771,659 filed Mar. 1, 2013, andU.S. Provisional Patent Application No. 61/905,536 filed Nov. 18, 2013,all of which are hereby incorporated by reference herein for allpurposes, as if fully set forth herein.

U.S. patent application Ser. No. 14/195,743 incorporated by referenceU.S. patent application Ser. No. 13/802,240 filed Mar. 13, 2013, whichis also hereby incorporated by reference herein for all purposes, as iffully set forth herein.

TECHNICAL FIELD

The present disclosure relates to human-computer interaction generallyand more specifically to gaze detection.

BACKGROUND

Human-computer interaction generally relates to the input of informationto and control of a computer by a user. Many common and popular computerprograms and operating systems have been developed to function primarilywith input methods involving physical contact or manipulation (e.g., amouse or a keyboard). This type of physical input method is referred toherein as contact-required input. It can be difficult for people whodesire to use non-contact input methods to interact with these computerprograms and operating systems to their full potential. Some people mustuse non-contact input methods for various reasons (e.g., because of aninjury or disability).

An example of a non-contact input device is an eye tracking device suchas that described in U.S. Pat. No. 7,572,008. Eye tracking devices canoperate on the principal of illuminating an eye with infrared light andutilizing an image sensor to detect reflection of the light off the eye.A processor can use the image sensor's data to calculate the directionof a user's gaze.

However, as technology progresses, computer programs and operatingsystems incorporate new forms of human-computer interaction, based oncontact-required inputs, to enable both simple and complexfunctionality. An example of a form of human-computer interaction istouch-based interaction on a computer, tablet, phone or the like,whereby a user interacts with the device by touching and by performinggestures (e.g., multi-finger gestures) on a touch-sensitive device(e.g., a touchscreen). This and other forms of user interaction requirea very physical connection between the device and the user, oftenrequiring multiple points of physical contact between the user and thetouch-sensitive device (e.g., for multi-finger gestures).

It can be desirable to develop human-computer interaction methods basedon non-contact inputs with the ability to perform both simple andcomplex functionality. It can be further desirable to develophuman-computer interaction methods based on non-contact inputs that canfunction effectively on computing devices developed for use primarilywith contact-required inputs.

Many non-contact interactions lack the clear definition andidentification of contact methods, therefore it can sometimes beambiguous as to the intention of a non-contact input command. In orderto assist with this problem, it has previously been proposed to utilizea non-contact input such as eye-tracking with a contact-required inputdevice, such as a computer mouse or touchpad. For example, U.S. Pat. No.6,204,828 describes a system whereby display of a cursor on screen issuspended and displayed at a user's gaze location upon movement by acomputer mouse.

Some interaction methods are not intuitive and the user may not know forsure if the eye tracking is functioning or the exact location of thecursor. Some interaction methods result in a cognitive disruptionwhereby after the user has triggered a movement of a cursor, the usermust anticipate the future location of the cursor and adjustaccordingly.

It can be desirable to signal to the user as early as possible thefuture location of the cursor while determining whether the user intendson triggering a mouse movement. Further, as eye tracking systems may notprovide 100% accuracy, the determined gaze position to which a cursorwill move may not be the position intended by the user. It can bedesirable to assist a user with more accurately determining how and whento use a non-contact input, such as eye tracking, in combination with acontact-required input, such as a touchpad or mouse.

SUMMARY

The term “embodiment” and like terms are intended to refer broadly toall of the subject matter of this disclosure and the claims below.Statements containing these terms should be understood not to limit thesubject matter described herein or to limit the meaning or scope of theclaims below. Embodiments of the present disclosure covered herein aredefined by the claims below, not this summary. This summary is ahigh-level overview of various aspects of the disclosure and introducessome of the concepts that are further described in the DetailedDescription section below. This summary is not intended to identify keyor essential features of the claimed subject matter, nor is it intendedto be used in isolation to determine the scope of the claimed subjectmatter. The subject matter should be understood by reference toappropriate portions of the entire specification of this disclosure, anyor all drawings and each claim.

Embodiments of the present disclosure include computer systems that canbe controlled with non-contact inputs through zonal control. In oneembodiment, a non-contact input tracks a non-contact action performed bya user. A computer's display, and beyond, can be separated into a numberof discrete zones according to a configuration. Each zone is associatedwith a computer function. The zones and/or their functions can, but neednot, be indicated to the user. The user can perform the various computerfunctions by performing non-contact actions detected by the non-contactinput. Upon indicating a desired zone associated with a particularfunction, the user can provide an activation signal of intent. Theactivation signal of intent can be a contact-required or non-contactaction, such as a button press or dwelling gaze, respectively. Uponreceiving the activation signal of intent, the computer system can usethe indicated zone (e.g., indicated by the user's non-contact actions)to perform the function associated with that zone.

Embodiments of the present disclosure include a computer system that canbe controlled with non-contact inputs, such as eye-tracking devices. Avisual indicator can be presented on a display to indicate the locationwhere a computer function will take place (e.g., a common cursor). Thevisual indicator can be moved to a gaze target in response to continueddetection of an action (e.g., touchpad touch) by a user for apredetermined period of time. The delay between the action and themovement of the visual indicator provides an opportunity to provide anindication to the user where the visual indicator will be located aftera movement, allowing for less of a cognitive disruption after the visualindicator has appeared at a new location. Optionally, the delay can alsoallow a user time to “abort” movement of the visual indicator.Additionally, once the visual indicator has moved, the visual indicatorcan be controlled with additional precision as the user moves gaze whilecontinuing the action (e.g., continued holding of the touchpad).

Embodiments of the present disclosure include a computer system that canbe controlled with non-contact inputs, such as eye-tracking devices. Acomputer can enlarge a portion of a display adjacent a first gaze targetin response to detecting a first action (e.g., pressing a touchpad). Thecomputer can then allow a user to position a second gaze target in theenlarged portion (e.g., by looking at the desired location) and performa second action in order to perform a computer function at thatlocation. The enlarging can allow a user to identify a desired locationfor a computer function (e.g., selecting an icon) with greaterprecision.

Embodiments of the present disclosure include a computer system that canbe controlled with non-contact inputs, such as eye-tracking devices.Various combinations of non-contact actions and contact-required actionscan be performed to cause a computer to perform certain computerfunctions. Functions can include scroll functions, movements of visualindicators, zooming of the display, and selecting further functions toperform. Combinations of non-contact actions and contact-requiredactions can include pressing buttons and/or touching touch-sensitivedevices while looking at certain places on or off of a display.

BRIEF DESCRIPTION OF THE DRAWINGS

The specification makes reference to the following appended figures, inwhich use of like reference numerals in different figures is intended toillustrate like or analogous components

FIG. 1 is a schematic representation of a computer system incorporatingnon-contact inputs according to certain embodiments.

FIG. 2A is a graphical depiction of a display as rendered or presentedon the display device of FIG. 1 according to certain embodiments.

FIG. 2B is a graphical depiction of the display of FIG. 2A in zonalcontrol mode with a first configuration according to certainembodiments.

FIG. 3 is a flow chart depicting a process for zonal control accordingto certain embodiments.

FIG. 4 is a graphical depiction of a display as rendered or presented onthe display device of FIG. 1 while in zonal control mode with a secondconfiguration according to certain embodiments.

FIG. 5 is a graphical depiction of a display with a visual indicatoraccording to certain embodiments.

FIG. 6 is a flow chart diagram of delay warp as performed by a computeraccording to certain embodiments.

FIG. 7A is a flow chart depicting a multi-step click functionalityaccording to some embodiments.

FIG. 7B is a flow chart depicting a multi-step click functionalityaccording to some embodiments.

FIG. 8 is a graphical depiction of a display according to certainembodiments.

FIG. 9 is a graphical depiction of a display according to certainembodiments.

FIG. 10 is a graphical depiction of a menu according to certainembodiments.

FIG. 11 is a graphical depiction of a display according to certainembodiments.

FIG. 12 is a graphical depiction of a display according to certainembodiments.

FIG. 13 is a graphical depiction of a display according to certainembodiments.

FIG. 14 is a graphical depiction of a display according to certainembodiments.

FIG. 15 is a graphical depiction of a display according to certainembodiments.

FIG. 16 is a graphical depiction of a display according to certainembodiments.

FIG. 17 is a graphical depiction of a display according to certainembodiments.

FIG. 18 is a graphical depiction of a display according to certainembodiments.

FIG. 19 is a graphical depiction of a display according to certainembodiments.

FIG. 20 is a graphical depiction of a display according to certainembodiments.

FIG. 21 is a graphical depiction of a display according to certainembodiments.

FIG. 22A is a graphical depiction of a display according to certainembodiments.

FIG. 22B is a graphical depiction of the display of FIG. 22A showing amenu according to certain embodiments.

FIG. 22C is a graphical depiction of the display of FIG. 22C showing amenu according to certain embodiments.

FIG. 23 is a graphical depiction of a display according to certainembodiments.

FIG. 24A is a flow chart of a non-contact action according to certainembodiments.

FIG. 24B is a flow chart of a non-contact action according to certainembodiments.

FIG. 24C is a flow chart of a contact-required action according tocertain embodiments.

FIG. 24D is a flow chart of a non-contact action according to certainembodiments.

FIG. 25 is a flow chart of a delay warp 2500 with a visual markeraccording to certain embodiments.

FIG. 26 is a flow chart of a delay warp 2600 without a visual markeraccording to certain embodiments.

FIG. 27 is a flow chart of a delay warp 2700 without a hidden visualindicator according to certain embodiments.

FIG. 28 is a flow chart of a delay warp 2800 according to certainembodiments.

FIG. 29 is a flow chart depicting a two-step click 2900 according tocertain embodiments.

FIG. 30 is a flow char depicting a method of providing input to a userinterface using a gaze interaction.

DETAILED DESCRIPTION

A computer system can be controlled with non-contact inputs throughzonal control. In an embodiment, a non-contact input that is aneye-tracking device is used to track the gaze of a user. A computer'sdisplay can be separated into a number of discrete zones according to aconfiguration. Each zone is associated with a computer function. Thezones and/or their functions can, but need not, be indicated to theuser. The user can perform the various functions by moving gaze towardsthe zone associated with that function and providing an activationsignal of intent. The activation signal of intent can be acontact-required or non-contact action, such as a button press ordwelling gaze, respectively.

A computer system can implement a delay warp when being controlled withnon-contact inputs. In an embodiment, a cursor can be presented on adisplay to indicate the location where a computer function will occurupon a further action (e.g., a click). The cursor can be moved to a gazetarget in response to continued detection of an action (e.g., continuedtouching of a touchpad) by a user for a predetermined period of time.The delay between the action and the movement of the cursor provides anopportunity to provide an indication to the user where the visualindicator will be located after a movement, allowing for less of acognitive disruption after the visual indicator has appeared at a newlocation. Optionally, the delay gives a user an opportunity to “abort”movement of the cursor. Additionally, once the cursor has moved, thecursor can be further controlled with additional precision as the usermoves gaze, moves a mouse, or swipes a touchpad while continuing theaction (e.g., continued holding of the touchpad).

A computer system can allow for increased certainty and precision whentargeting elements through non-contact inputs. In an embodiment, a usercan look at a group of elements and perform an action. If the computercannot determine with certainty which element is targeted by the user,the computer can enlarge and/or separate the elements and allow the userto further focus gaze on the desired element, whereupon performing asecond action, the computer will perform the desired function (e.g.,selecting an icon) upon the targeted element.

A computer system can be controlled through various combinations ofnon-contact actions and contact-required actions. Scrolling, cursormovements, zooming, and other functions can be controlled throughcombinations of non-contact actions and/or contact-required actions.Such combinations can include pressing buttons and/or touchingtouch-sensitive devices while looking at certain places on or off of adisplay.

These illustrative examples are given to introduce the reader to thegeneral subject matter discussed here and are not intended to limit thescope of the disclosed concepts. The following sections describe variousadditional features and examples with reference to the drawings in whichlike numerals indicate like elements, and directional descriptions areused to describe the illustrative embodiments but, like the illustrativeembodiments, should not be used to limit the present disclosure. Theelements included in the illustrations herein may be drawn not to scale.As used herein, examples listed with the use of exempli gratia (“e.g.”)are non-limiting examples.

FIG. 1 is a schematic representation of a computer system 100incorporating non-contact inputs 106 according to certain embodiments.The computer system 100 (hereinafter, “computer”) can be implemented ina single housing (e.g., a tablet computer), or can be implemented inseveral housings connected together by appropriate power and/or datacables (e.g., a standard desktop computer with a monitor, keyboard, andother devices connected to the main housing containing the desktopcomputer's CPU). As used herein, any reference to an element existing“in” the computer 100 indicates the element is a part of the computersystem 100, rather than physically within a certain housing.

The computer 100 can include a processor 102 connected to or otherwisein communication with a display device 104, a non-contact input 106, anda contact-required input 108. The processor 102 can include anon-contact interpreter 112, as described in further detail below. Asused herein, the term processor 102 refers to one or more individualprocessors within the computer system, individually or as a group, asappropriate. The computer 100 can include programming 116 stored onpermanent, rewritable, or transient memory that enables the processor102 to perform the functionality described herein, including zonalcontrol, delay warp, and two-step click, as well as other functionality.The programming (e.g., computer-executable instructions or other code),when executed by the processor 102, causes the processor 102 to performoperations described herein. The programming may compriseprocessor-specific programming generated by a compiler and/or aninterpreter from code written in any suitable computer-programminglanguage. Non-limiting examples of suitable computer-programminglanguages include C, C++, C#, Visual Basic, Java, Python, Perl,JavaScript, ActionScript, and the like. The memory may be acomputer-readable medium such as (but not limited to) an electronic,optical, magnetic, or other storage device capable of providing aprocessor with computer-readable instructions. Non-limiting examples ofsuch optical, magnetic, or other storage devices include read-only(“ROM”) device(s), random-access memory (“RAM”) device(s), magneticdisk(s), magnetic tape(s) or other magnetic storage, memory chip(s), anASIC, configured processor(s), optical storage device(s), floppydisk(s), CD-ROM, DVD, or any other medium from which a computerprocessor can read instructions.

Contact-required inputs 108 can be any device for accepting user inputthat requires physical manipulation or physical contact (hereinafter,“contact-required actions”). Examples of contact-required inputs 108include keyboards, mice, switches, buttons, touchpads, touchscreens,touch-sensitive devices, and other inputs that require physicalmanipulation or physical contact. Examples of contact-required actionsinclude tapping, clicking, swiping, pressing (e.g., a key), and others.As used herein, the term “contact-required actions” further includeactions that require either physical contact through another device(e.g., using a touchscreen with a stylus) or close proximity to thecontact-required input (e.g., hovering or swiping a finger above atouchscreen that responds to fingers in close proximity, such as aprojected capacitance touchscreen). Signals generated from a userperforming contact-required actions as received by contact-requiredinputs 108 are referred to as contact-based signals 110. Whereappropriate, references to contact-required actions can includecombinations of contact-required actions (e.g., holding a first buttonwhile pressing a second button).

Non-contact inputs 106 can be any device capable of receiving user inputwithout physical manipulation or physical contact. Examples ofnon-contact inputs 106 include eye-tracking devices, microphones,cameras, light sensors, and others. When a user performs an actiondetectable by a non-contact input 106 (hereinafter, “non-contactaction”), the non-contact interpreter 112 generates a non-contact signal114 based on the non-contact action performed by the user. Non-contactactions can include moving gaze (e.g., moving the direction of the gazeof one or more eyes), gaze saccade, fixating gaze, dwelling gaze (e.g.,fixating gaze substantially at a single target for a predeterminedlength of time), a blink (e.g., blinking one or more eyes once or in adiscernible pattern, or closing one or both eyes for a longer length oftime), performing vocal commands (e.g., saying “click” or “open”),facial recognition (e.g., recognizing features and movements of theuser's face), 3-D gestures (e.g., recognizing movements of a user, auser's appendage, or an object held by the user in 3-D space, such aswaving), and others. Depending on the action performed, the non-contactinterpreter 112 can send different non-contact signals 114. For example,a user moving gaze can result in a first non-contact signal 114containing information about the movement and/or new direction of gaze,while a user blinking can result in a non-contact signal 114 indicativethat the user blinked. The non-contact signal 114 can be used by theprocessor 102 to perform various tasks, as described in further detailbelow. Where appropriate, references to non-contact actions can includecombinations of non-contact actions (e.g., blinking while saying“click”).

Additionally, where appropriate, references to an action can includecombinations of contact-required actions and non-contact actions (e.g.,holding a button while saying “click”).

In some embodiments, the processor 102 can use non-contact signals 114to emulate contact-required signals 110. For example, the processor 102can use non-contact signals 114 containing information about a usermoving gaze to a first target (e.g., computer icon) and dwelling gaze onthat target in order to emulate contact-required signals 110 of moving acursor to the first target and clicking on that target.

The embodiments disclosed herein include the use of non-contact actionsand contact-required actions, or non-contact actions alone, to performvarious computer functions. Computer functions can be any type of actionperformable on a computer, such as a mouse click; a scroll/pan action; amagnifying action; a zoom action; a touch input/action; a gestureinput/action; a voice command; a call-up of a menu; the activation ofEye Tracking/Eye Control/Gaze interaction; the pausing of EyeTracking/Eye Control/Gaze interaction; adjusting the brightness of thebacklight of the device; activating sleep, hibernate, or other powersaving mode of the device; resuming from sleep, hibernate, or otherpower saving mode of the device; or others. In some cases, the computerfunctions are emulated such that the computer 100 behaves as if it isdetecting solely a contact-required action.

Zonal Control

FIG. 2A is a graphical depiction of a display 200 as rendered orpresented on a display device 104 according to certain embodiments.While presented with reference to the particular operating system shownin FIG. 2A, the embodiments and disclosure herein can be easily adaptedto other operating systems (e.g., Microsoft Windows®, Apple iOS®, orGoogle Android®) and computer programs to perform various functionsthereof. The operating system of FIG. 2A includes various computerfunctionality available through contact-required inputs 108 (e.g.,touchscreen taps and gestures). Table 1, below, describes several ofthese functions and examples of associated contact-required actions toperform each function.

TABLE 1 Computer Function Example of Associated Contact-required ActionOpen Start Menu/ Swipe in from the right edge of the display 200 and tapStart Screen a “Start” button that appears. Open Apps List Swipe up fromthe center of the Start screen Previous App Swipe in from the left edgeof the display 200 Display Charms Swipe in from the right edge of thedisplay 200 Open App Bar Swipe in from the top or bottom edge of thedisplay 200 Move App Bar Tap and drag the app bar Hide App Bar Tap andhold and swipe down the app bar Split Window Tap and drag the app fromthe top edge of the display 200 towards the middle of the display 200,and then towards either the left or right edge of the display 200. CloseApp Tap and drag the app from the top edge of the display 200 to thebottom of the display 200

In order to accommodate performing these and other computer functionswithout relying solely on contact-required actions, a computer 100 canuse detected non-contact actions (e.g., movement of gaze and fixation ontarget) as instruction to perform various computer functions. Thecomputer can perform a computer function instigated by a non-contactaction by either performing the computer function directly (e.g.,opening the apps list) or by emulating the contact-required actionassociated with the computer function (e.g., emulating a swipe up fromthe center of the Start screen).

FIG. 2B is a graphical depiction of the display 200 of FIG. 2A in zonalcontrol mode 206 according to certain embodiments. The display 200 isseparated into a first configuration 204 of eight zones 202, however anynumber of zones 202 and any different configuration can be used. Thezones 202 and/or lines between the zones 202 can, but need not, bepresented to the user on the display device 104 (e.g., highlighting of azone 202 or lines separating the zones 202). Zones 202 are used toenable non-contact control of the computer 100. Each zone 202 can beassociated with a particular computer function. In some embodiments, thezones 202 are divided and/or located such that a “dead zone” of nofunctionality exists between some or all of the zones 202, so ensurethat measurement errors and/or data noise do not cause undesiredeffects. In some embodiments, hysteresis can be used to avoidinadvertently selecting an undesired function (e.g., by increasing theboundaries of a zone 202 while the gaze target is in the zone 202 or byintroducing a certain amount of delay when the gaze target moves out ofa particular zone 202, before altering any performed function).

The computer function associated with each zone 202 can, but need not,be presented to the user on the display device 104 (e.g., a text box).The computer 100 can include a non-contact input 106 that is aneye-tracking device. An eye-tracking device can detect eye indicationsof a user. As used herein, the term “eye indications” is inclusive ofdetecting the direction of a user's gaze, detecting changes in thedirection of a user's gaze (e.g., eye movement), detecting blinking ofone or both eyes, and detecting other information from a user's eye oreyes. A non-contact target 208 is a computed location of where a user'snon-contact action is directed. The non-contact target 208 can begraphically represented on the display, such as by a symbol shown inFIG. 2. In the example of an eye-tracking device, the non-contact target208 is a gaze target, or the point where the user's gaze is directed.The non-contact target 208 can be indicated by 3-D gestures, facialorientation, or other non-contact actions. The non-contact target 208can, but need not, be depicted in some fashion on the display 200 (e.g.,presenting a symbol at the non-contact target or highlighting elementsor zones at or near the non-contact target).

In some embodiments, a zone 202 can be located outside of the display200 and the non-contact target 208 need not be constrained to thedisplay 200 of the computer 100. For example, a zone 202 can be locateda distance to the left of a display device 104 and can be associatedwith a certain computer function. The user can perform the function inthat zone 202, as described in further detail below, by focusing gaze tothe left of the display device 104. Determination of a zone 202 outsideof the display 200 can occur via an imaging device forming part of aneye tracking device or a separate imaging device. If a user's gaze isdetermined to be directed towards an area outside of the display 200,the direction of the gaze can be determined as herein described and ifthe gaze target falls within the bounds of a zone 202 outside of thedisplay 200, an appropriate function can be performed as furtherdescribed herein.

In some embodiments, statistical analysis can be applied to the detectedgaze target and/or detected movements of the gaze target in order todetermine whether the gaze target is in a particular zone 202.

In some embodiments, a lockout time can be implemented whereby if a useractivates a function associated with a zone 202, the function associatedwith the zone 202 cannot be activated (e.g., activated in the samemanner or in a different manner) until the expiration of a certainlength of time. In some embodiments, after a user activates a functionassociated with a zone 202, the size of the zone 202 decreases for aperiod of time such that a more deliberate gaze by the user is requiredto activate the function associated with that particular zone 202 again.

FIG. 3 is a flow chart depicting a process 300 for zonal controlaccording to certain embodiments. With reference to FIGS. 2B and 3, oneembodiment of zonal control is discussed below.

The user can generate a mode-enable signal of intent 310. Themode-enable signal of intent 310 can be generated by the user performinga non-contact action or a contact-required action. The computer 100detects the mode-enable signal of intent 310 at block 302 and enters azonal control mode 206. During zone control mode 206, the computer 100tracks the non-contact target 208 at block 304. When the computer 100detects an activation signal of intent 312 at block 306, the computer100, at block 308, then performs a computer function associated with thezone 202 in which the non-contact target 208 is located at the time ofthe activation signal of intent 312. The activation signal of intent 312can, but need not, be the same type of signal of intent as themode-enable signal of intent 310. Examples of signals of intent includecontact-required actions and non-contact actions. In some embodiments,the computer 100 is always in a zonal control mode 206 (i.e., noseparate mode-enable signal of intent 310 is necessary), whereuponreceiving an activation signal of intent 312, the computer 100 willperform the function associated with the zone 202 in which thenon-contact target 208 is located at the time of the activation signalof intent 312. The computer 100 can, but need not, provide visualfeedback that an activation signal of intent 312 was received, that thenon-contact target 208 was in a particular zone 202, and/or that aparticular function was activated.

In a first example, a user can speak out loud “zone mode” (i.e., performa non-contact action) to generate the mode-enable signal of intent 310.Upon detecting this mode-enable signal of intent 310, the computer 100enters zonal control mode 206. The user can then focus gaze somewhere inthe zone 202 associated with the computer function Open Apps List 210.The user can dwell gaze (i.e., perform a non-contact action) over thezone 202 associated with the computer function Open Apps List 210 for apredetermined amount of time (e.g., 3 seconds) to generate an activationsignal of intent 312. The computer 100 can detect that the user isdwelling gaze at least by detecting that the non-contact target 208 thatis a gaze target dwells at a location (e.g., does not move substantiallyor moves only as much as expected for a user attempting to look at thesame location) for a predetermined amount of time. Upon detecting theactivation signal of intent 312, the computer 100 can perform the OpenApps List 210 function (e.g., by directly performing the function or bysimulating a touchscreen swipe up from the center of the Start screen).The computer 100 can then exit out of zonal control mode 206.

In a second example, a user can focus gaze somewhere in the zone 202associated with the computer function Display Charms 222. The user candepress a hardware button (i.e., perform a contact-required action) togenerate an activation signal of intent 312. Upon detecting theactivation signal of intent 312, the computer 100 can perform theDisplay Charms 222 function. In this second example, no mode-enablesignal of intent 310 is necessary.

Graphical representations of the zones 202 may disappear upon an actionby a user, or after a predetermined period of time. In the absence ofgraphical representations of the zones 202, the zonal control mode 206still functions as described

Some examples of possible computer functions associated with potentialzones 202 include Open App Bar 214, Move App Bar 216, Hide App Bar 218,Previous App 220, Split Window/Close App 224, and others.

A signal of intent can be any non-contact action detectable by thecomputer 100. A signal of intent can be any contact-required actiondetectable by the computer 100. In some embodiments, a signal of intentcan be selection and/or activation of an icon in an input menu.

In some embodiments, the shape of the zones 202 and computer functionsassociated with each zone 202 can change depending on the state of thecomputer 100. For example, upon using zonal control to perform thecomputer function Open Apps List 210, the new window that appears caninclude different zones 202 with different computer functions associatedtherewith. FIG. 4 is a graphical depiction of a display 400 as renderedor presented on the display device 104 of FIG. 1 while in zonal controlmode 206 with a second configuration 402 according to certainembodiments. The display 200 is separated into a second configuration402 of zones 202. The second configuration 402 of zones 202 can beassociated with the state of the computer 100 after the Open Apps List210 function has been performed. In this configuration, there are zones202 associated with various computer functions, including start 212,display charms 222, scroll right 404, scroll left 406, zoom out 408, andzoom in 410. The second configuration 402 need not be dependent on a newscreen being displayed on the display 200, but can be associated withany state of the computer 100. For example, the zoom in 410 and scrollleft 406 zones 202 may not be a part of the second configuration 402until needed (e.g., a zoom out 408 or scroll right 404 function has beenperformed, respectively). The zones 202 in the second configuration 402can otherwise perform similarly to the zones 202 of the firstconfiguration 204.

In some embodiments, zones 202 can overlap, such that multiple computerfunctions are performed simultaneously from activation of the zones 202when the gaze target is within both zones 202. For example, if theoverlapping zones 202 were associated with scrolling functions, such asa scroll up function and a scroll right function, activation of thezones 202 (e.g., by moving a gaze target into the zones) can result inthe window scrolling diagonally up and to the right.

FIGS. 8-10 demonstrate an embodiment whereby a zone is in the form of amenu overlaid atop a computer display. The menu may appear or disappearbased on a contact or non-contact input action. The menu comprisesoptions for selection representing computer functions. The options maybe selected by gazing at an item and providing an activation signal ofintent. By way of example, a user may gaze at an item representing acommon computer function known as a “left click”, the user may fixate atthe item thus providing the activation signal of intent. Once theactivation signal of intent has been provided, the computer will performa “left click” at the next location at which the user fixates orprovides another activation signal of intent. In this manner, the usermay select the function he or she desires to execute, and then selectthe location upon the display at which the function is to be executed.

In some embodiments, a user can look at a first zone, then look at alocation away from the first zone to perform a computer functionassociated with the zone in which the non-contact target was located.For example, a user can look at a menu item as described above, initiatean activation signal of intent, then look at an icon, and then initiatea second activation signal of intent. In this example, the computer 100can determine the function to be performed on the icon based on wherethe user's gaze was located (i.e., at the zone).

Any function possible on a computer can be performed (e.g., directlyperformed or emulated) through the use of zonal control as describedherein. In the case of computers designed primarily for touchscreeninput (e.g., certain mobile computing devices such as mobile phones andtablets), performed functions can include those such as opening anapplication, navigating the display (e.g., navigating to a new page byswiping a finger across the display), zooming, hardware buttons (e.g., ahome button or return button), multi-finger gestures, and others. Somesingle and multi-finger gestures can include tap, double-tap, longpress, press and hold, touch and hold, slide, swipe, drag, press anddrag, fling, flick, turn, rotate, scroll, pan, pinch, stretch, edgeswipe/bezel swipe, and others. In the case of computers designed forother contact-required input (e.g., desktop computers with a keyboardand mouse), zonal control can be used to perform functions includingmouse movement, mouse clicking (e.g., single click, double click, rightclick, or click and drag), keyboard presses, keyboard combinations, orother functions related to contact-required actions.

Delay Warp

The computer 100 can be controlled through one or both of thenon-contact input 106 and the contact-required input 108.

FIG. 5 is a graphical depiction of a display 500 with a visual indicator502 according to certain embodiments. The visual indicator 502 is usedlike a mouse cursor to help a user perform computer functions (e.g.,clicking, dragging, and others). The visual indicator 502 is anindication of where the effect of an additional action (e.g., a mouseclick, a touchpad tap, or other contact-required or non-contact action)will occur. The computer 100 can generate a visual indicator 502 on thedisplay at an estimated location of the non-contact target 208, asdescribed above.

In one embodiment, the visual indicator 502 can be displayed at anestimated gaze target of the user. The estimated gaze target iscalculated by an eye-tracking device or by a computer 100 usinginformation from an eye-tracking device. The computer 100 containsprogramming 116 enabling the processor 102 to perform a delay warp, asdescribed below.

FIG. 6 is a flow chart diagram of delay warp 600 as performed by acomputer 100 according to certain embodiments. A computer 100 canoptionally perform a click according to input from a contact-requiredinput, such as a computer mouse or a touchpad, at block 602. A user canperform an action which is detected at block 606. In one embodiment,upon detecting an action at block 606, the computer 100 can display avisual marker at an estimated gaze target of the user. The visual markercan be an indicator of to where the cursor will move as describedherein. For example, the user can perform a contact-required action,such as touching a touch-sensitive device. Upon detecting the action atblock 606, the computer 100 can delay, for a predetermined amount oftime, at block 610. This delay can be utilized by the computer 100 toprovide sufficient time for a user to alter action (e.g., decide not tomove the cursor) and for the computer 100 to be certain of the user'sintention. At block 612, if the computer 100 still detects the action(e.g., the user continues to touch the touch-sensitive device), thecomputer 100 can move the visual indicator 502 to the gaze target atblock 614. If the action is not still detected, the computer 100 can, atblock 602, do nothing, go back to having a contact-required input movethe visual indicator 502 (e.g., cursor), or perform a click at or movethe visual indicator 502 to its original location.

In an embodiment, the delay warp 600 additionally includes optionalblock 618 where the computer 100 determines whether the action is stilldetected (i.e., after the visual indicator 502 has moved to the gazetarget). If the action is not still detected, such as if the user is nolonger touching the touch-sensitive device, the computer 100 can performadditional functions as necessary (e.g., perform a “click” where thevisual indicator 502 was last located, or do nothing) at path 620.However, if the action is still detected at optional block 618, thecomputer 100 can slowly move (e.g., with more precision) the visualindicator 502 (e.g., a cursor) according to movements of the user'sgaze, a computer mouse, or a touchpad at optional block 622.

For example, if a user desires to select an element 504 on a display500, such as an icon, the user can look at the icon and touch thetouchpad for the predetermined period of time, such as 200 ms, afterwhich the computer 100 will move the visual indicator 502 to the icon.If the user, before the predetermined period of time has elapsed,decides not to have the visual indicator 502 moved to the gaze target,the user can cease touching the touchpad.

In an embodiment, the user can touch the touchpad or click a mousebutton and wait the predetermined period of time so that the visualindicator 502 moves to the gaze target. Thereafter, the user cancontinue touching the touchpad or holding the mouse button while movinggaze away from the visual indicator 502 (e.g., above, below, or to theside) in order to move the visual indicator 502 with fine-tuneadjustments until the visual indicator 502 is in a desired location, atwhich point the user can cease touching the touchpad or holding themouse button in order to perform an action at the desired location(e.g., click an icon).

In some embodiments, the user can touch the touchpad or click a mousebutton and wait the predetermined period of time so that the visualindicator 502 moves to the gaze target. Thereafter, the user cancontinue touching the touchpad or holding the mouse button while movingthe cursor with the touchpad or mouse in order to move the visualindicator 502 (e.g., cursor) with fine-tune adjustments until the visualindicator 502 is in a desired location, at which point the user cancease touching the touchpad or holding the mouse button in order toperform an action at the desired location (e.g., click an icon).

A user can look at a desired screen element 504 (e.g., an icon, awindow, or other graphical user interface (“GUI”) element) in order todirect the visual indicator 502 to that element 504. In order to performa desired computer function (e.g., a click), the user can perform anadditional action (e.g., tap a touchpad).

In some embodiments, the visual indicator 502 may not be regularlydisplayed, but as the user moves gaze around the display 500, anyelements 504 at or adjacent the gaze target can be highlighted orotherwise distinguish the element 504 as at or near the gaze target.

In some embodiments, the use of a visual indicator 502 enables a user tosee the effect of non-contact actions on the computer 100 beforeperforming additional actions (e.g., non-contact actions orcontact-required actions). When a user intends to move the visualindicator 502 or other graphical element 504 on a display, the userlooks at the desired destination of the visual indicator 502. Theeye-tracking device calculates an estimated gaze target based on theuser's gaze. The user then activates a non-contact input 106 or acontact-required input 108, for example by tapping a touchpad. For apredetermined period of time, for example 200 ms, the computer 100 doesnot perform a computer function.

During this predetermined time, the visual indicator 502 is shown on thedisplay 500 at the estimated gaze target. This visual indicator 502 or aseparate visual marker can then demonstrate to the user the location towhich the visual indicator 502 will be moved. If the user determines toproceed, the visual indicator 502 will be moved after a predeterminedperiod of time. The user can indicate a desire to proceed by initiatingan action (e.g., a contact-required action such as moving an inputdevice) or by simply waiting for the predetermined period of time toexpire.

If the user determines not to proceed, the user may perform a specificaction such as removing contact with the input device, tapping the inputdevice or pressing and holding the input device. Typically, theseactions cause the computer 100 to perform a specific function, such astapping to open an icon, dragging such as dragging an item on a GUI,zooming upon an item, or others. Actions that are normally performedwith an input device would be readily understood by a person skilled inthe art.

If the user is not satisfied with the location of the visual indicator502, the user can determine that an adjustment is required in order tomore accurately reflect the desired movement location of the visualindicator 502. The user can gaze at a different location in order tochange the gaze target, or the user can perform a small movement with aninput device (e.g., move a computer mouse or move touch on a touchpad)to adjust the location of the visual indicator 502 after the visualindicator 502 has moved to the gaze target.

In this manner, natural interaction with a touchpad accommodates gazeinformation. If a user places their finger on a touchpad in order toperform a gesture such as a swipe or movement on the touchpad, themovement can override movement of the mouse cursor to the gaze location.

In one embodiment, a computer 100 includes a touchpad as acontact-required input device and an eye tracking device capable ofdetermining a user's gaze target.

The computer 100 utilizes input from both the touchpad and the eyetracking device to allow a user to navigate through user interfaces.Most frequently this is achieved through moving a visual indicator 502on a display 500.

The computer 100 utilizes gesture type commands used on the touchpad,for example a swipe across the touchpad by a user to move to the nextelement 504 in a series, or a pinch gesture on the touchpad to zoom adisplayed element 504.

According to the present disclosure, when a user contacts the touchpadwith a finger or the like, the computer 100 delays performing a computerfunction for a predetermined period of time. During this period of time,a visual indicator 502 is shown at an estimated gaze target of the user.The estimated gaze target is calculated based on information from theeye tracking device.

After the predetermined period of time expires, the computing systemmoves the location of the visual indicator 502 on the display 500 to thegaze target. The user can then move the visual indicator 502 by movingtheir finger on the touchpad.

If the user does not wish for the visual indicator 502 to be moved tothe gaze target, the user can perform another action during thepredetermined period of time—such as the aforementioned gesture typecommands, or simply remove their finger from the touchpad to cancel anyaction.

In an embodiment, the computer 100 can locate the visual indicator 502at an element 504 in proximity to the actual gaze target. This element504 can be a Graphical User Interface (GUI) object, for example, such asa button, text box, menu or the like. The computer 100 can determinewhich element 504 at which to locate the visual indicator 502 based on aweighting system whereby some elements 504 have predetermined weightshigher than other elements 504. For example, a button can have a higherweighting than a text box. The determination of which element 504 atwhich to place the visual indicator 502 can also consider proximity tothe gaze target.

In an embodiment, the computer 100 can provide tactile or audiblefeedback indicating that a gaze target is able to be determined. In thisway, the feedback will indicate to the user whether the system isfunctioning correctly and if not, it will allow the user to alter theirbehavior to accommodate the function or non-function of the eye trackingdevice. This feedback can be in the form of a touchpad providing hapticfeedback when an estimated gaze position has been determined during acursor movement procedure.

FIG. 25 is a flow chart of a delay warp 2500 with a visual markeraccording to certain embodiments. The visual marker is an indication ofto where the visual indicator (e.g., cursor) might jump or “warp” undercertain conditions. At block 2502, a contact-required action isdetected, such as a user touching a touchpad. The computer 100 waits forthe duration of a predefined length of time (e.g., delay) at block 2504.In some embodiments, the delay can be 0 seconds (i.e., no delay). Afterthe delay, at block 2506, the computer 100 causes a visual marker to bedisplayed at the non-contact target (e.g., gaze target). In alternateembodiments, an additional delay can be incorporated after block 2506and before block 2508. At block 2508, the computer 100 determines if thecontact-required action is maintained, such as if the user is stilltouching the touchpad. If the contact-required action is maintained, thecomputer 100, at block 2510, then ceases to display the visual markerand moves the visual indicator (e.g., cursor) to the non-contact target.If the contact-required action is not maintained at block 2508, thecomputer 100 can cease displaying the visual marker at block 2512 andexecute a click at the location of the visual indicator (e.g., cursor)at block 2514.

FIG. 26 is a flow chart of a delay warp 2600 without a visual markeraccording to certain embodiments. At block 2602, a contact-requiredaction is detected, such as a user touching a touchpad. The computer 100waits for the duration of a predefined length of time (e.g., delay) atblock 2604. After the delay, at block 2606, the computer 100 moves thevisual indicator (e.g., cursor) to the non-contact target (e.g., gazetarget). In alternate embodiments, an additional delay can beincorporated after block 2606 and before block 2608. At block 2608, thecomputer 100 determines if the contact-required action is maintained,such as if the user is still touching the touchpad. If thecontact-required action is maintained, the process is finished at block2610. If the contact-required action is not maintained at block 2608,the computer 100, at block 2612, can move the visual indicator back toits original position prior to the movement from block 2606. Next, thecomputer 100 can execute a click at the location of the visual indicatorat block 2614.

FIG. 27 is a flow chart of a delay warp 2700 without a hidden visualindicator according to certain embodiments. At block 2702, the visualindicator is in a hidden state. Block 2702 includes all instances wherethe visual indicator is hidden, including if it has never been displayedpreviously. At block 2704, a contact-required action is detected, suchas a user touching a touchpad. The computer 100 waits for the durationof a predefined length of time (e.g., delay) at block 2706. After thedelay, at optional block 2708, the computer 100 can display either avisual marker or the visual indicator (e.g., cursor) at the non-contacttarget (e.g., gaze target). In alternate embodiments, an additionaldelay can be incorporated after block 2708 and before block 2710. Atblock 2710, the computer 100 determines if the contact-required actionis maintained, such as if the user is still touching the touchpad. Ifthe contact-required action is maintained, the computer 100 moves thevisual indicator to the non-contact target at block 2712. If thecontact-required action is not maintained at block 2708, the computer100, at block 2714, executes a click at the non-contact target.

FIG. 28 is a flow chart of a delay warp 2800 according to certainembodiments. At block 2802, a contact-required action is detected, suchas a user touching a touchpad. The computer 100 waits for the durationof a predefined length of time (e.g., delay) at block 2804. At block2806, the computer 100 determines whether the contact-required actionincludes signals to move the visual indicator (e.g., cursor). Suchactions can include swiping a finger along a touchpad or moving a mouse.If the contact-required action includes signals to move the visualindicator (e.g., the user touches a touchpad and moves the fingeraround), the computer 100, at block 2808, moves the visual indicatorpursuant to the contact-required action. If the contact-required actiondoes not include signals to move the visual indicator (e.g., the usertouches a touchpad and does not move the finger around), the computer,at optional block 2810, can show a visual marker or the visual indicatorat the non-contact target (e.g., gaze target), but then determines, atblock 2812, whether the contact-required action is maintained (e.g., theuser touches and keeps touching the touchpad). If the contact-requiredaction is not maintained, the computer 100, at block 2814, executes aclick at the original location of the visual indicator. If thecontact-required action is maintained, the computer 100, at block 2816,moves the visual indicator to the non-contact target. Then, at block2818, the computer 100 determines whether the contact-required actionnow includes signals to move the visual indicator (e.g., the user, afterseeing the visual indicator move to the non-contact target, beginsmoving the finger around on the touchpad). If the contact-requiredaction does not now include signals to move the visual indicator, thecomputer finishes the process at block 2822. However, if thecontact-required action does now include signals to move the visualindicator, the computer 100 moves the visual indicator pursuant to thenew contact-required action at block 2820. Optionally, upon release ofthe contact-required action, a “click” or other function may beperformed. In some embodiments, the movement of the visual indicator atblock 2820 is slower than the movement of the visual indicator at block2808.

After the delay, at optional block 2808, the computer 100 can displayeither a visual marker or the visual indicator (e.g., cursor) at thenon-contact target (e.g., gaze target). At block 2810, the computer 100determines if the contact-required action is maintained, such as if theuser is still touching the touchpad. If the contact-required action ismaintained, the computer 100 moves the visual indicator to thenon-contact target at block 2812. If the contact-required action is notmaintained at block 2808, the computer 100, at block 2814, executes aclick at the non-contact target.

Two-Step Click

A user can perform non-contact actions detectable by a computer 100through a non-contact input 106, such as an eye-tracking device. In someembodiments, a user can direct gaze at an elements 504 on a display 500and then perform an additional action in order to perform a computerfunction (e.g., a click) upon the element 504 upon which the user's gazeis directed. The computer 100 may not display any visual indication ofthe location of a user's gaze target. Although embodiments describedherein refer to a single “additional action” following an initialnon-contact action, is should be appreciated that multiple “additionalactions” (e.g., a sequence of keystrokes or any sequential orsimultaneous combination of non-contact and/or contact-required actions)may be required to trigger a particular computer function in otherembodiments.

FIG. 7A is a flow chart depicting a multi-step click functionality 700according to some embodiments. In a general embodiment, the computer 100detects a user's gaze target at block 702. The user's gaze target can belocated adjacent to or away from a display. The computer 100 thendetects a contact-required action (e.g., a button press or a touchpadtap) at block 704. The computer 100 then performs a computer function atblock 706, dependent on the gaze target and the contact-required action.

In one embodiment, no visual indication is displayed to the user untilafter block 704, upon which the computer 100 highlights the element 504located at or near the gaze target.

FIG. 7B is a flow chart depicting a multi-step click functionality 710according to some embodiments. The computer 100 detects a user's gazetarget at block 702. The computer 100 detects a first action (e.g., abutton press or a touchpad tap) at block 704. At block 712, the computer100 then determines whether there are multiple small elements 504located sufficiently close together adjacent the gaze target. In someinstance, multiple elements 504 include instances where each element 504is sufficiently close to one another that the computer 100 determinesadditional accuracy is necessary in order for the user to select thedesired element 504. At block 714, the computer presents a zoomed imageof a portion of the display near the user's gaze target. The portion ofthe display can be solely the elements 504 or can be the elements 504and surrounding aspects of the display (e.g., backgrounds). The computer100 continues to detect the user's gaze target at block 716. Optionally,the computer 100 can additionally highlight any element 504 selected bythe user's gaze target at block 716. The computer 100 can detect asecond action at block 718. Upon detecting the second action, thecomputer 100 can optionally highlight the element at the gaze target atblock 708 and then perform a computer function dependent upon theelement at the gaze target at block 706 (e.g., selecting the elementlocated at the gaze target). In situations where the computer 100 doesnot detect multiple small elements at block 712, the computer can simplycontinue directly to optional block 708 and block 706.

In this way, a computer 100 can dynamically adapt to a user's desire toselect small or numerous elements on a display using non-contactactions. If a user attempts to select small or numerous elements usingnon-contact actions, such as through eye-tracking, the computer 100 candynamically zoom in to allow the user to have better control for pickingthe correct element. As described above, the first actions and secondactions can be contact-required or non-contact actions.

In some embodiments, the first and second actions detected at blocks 704and/or 718 can be the pressing of a button or touching of a touchpad. Insome embodiments, the first and second action detected at blocks 704and/or 718 can be releasing a button that has already been pressedand/or ceasing to touch a touchpad that has previously been touched. Forexample, a user can depress a button while moving gaze around a displayand release the button when it is desired that the computer functiontake place. In some embodiments, the second action is a release of abutton while the first action is a depression of the same button.Additionally, the first action and second action are generally the sametype of actions (e.g., a button press), but need not be.

In some embodiments, the computer 100 can highlight the gaze targetand/or the elements at or near the gaze target while the button isdepressed. When the user looks at a group of elements that aresufficiently small and close together that the computer 100 will zoominto them (as described above with reference to block 714), the computer100 can highlight the group of elements, instead of a single element.

In an embodiment, the computer 100 can zoom in on the display when auser initiates an action (e.g., a contact-required action). The computer100 can zoom in on the display without first determining whether thereare multiple small elements located near the gaze target. The computer100 can otherwise function as described above with reference to FIG. 7B.

In one embodiment, the computer 100 can determine which element 504 atwhich to locate the gaze target based on a weighting system whereby someitems have predetermined weights higher than other items. For example, abutton can have a higher weighting than a text box. The determination ofwhich element 504 at which to place the cursor can also considerproximity to the estimated gaze position.

In another embodiment, the computer 100 can provide tactile or audiblefeedback indicating that an estimated gaze position is able to bedetermined. In this way, the feedback will indicate to the user whetherthe system is functioning correctly and if not it will allow the user toalter their behavior to accommodate the function or non-function of theeye tracking device. This feedback can be in the form of a touchpadproviding haptic feedback when an estimated gaze position has beendetermined during a cursor movement procedure.

FIG. 29 is a flow chart depicting a two-step click 2900 according tocertain embodiments. A computer 100 can determine a non-contact target(e.g., gaze target) at block 2902 and detect a first action at block2904. The first action can be a contact-required action (e.g., a buttonclick) or a non-contact action. Optionally, multiple elements providedon a display can be highlighted at once as the computer 100 determines anon-contact target is located on or near the elements. At block 2906,the computer 100 determines if an element at the non-contact target canbe reliably determined. In other words, the computer 100 determineswhether it is able to determine, with enough certainty, which elementthe user is intending to target. The computer 100 can consider multipleparameters to determine which element or elements the user is intendingto target. These parameters can be pre-set and/or user-set, but comprisetwo decision points. Firstly, the computer 100 determines expectedtracking deviation based on factors that can include user preference setby the user. Such preferences can include, for example, whether the userdesires speed or accuracy, user eye tracking quality including whetherthe user is wearing glasses, contact lenses or the like, the quality ofa calibration function mapping the user's gaze, estimates of offsets ofthe detected user's gaze relative to the actual location of the user'sgaze, signal noise in the eye tracking system, configuration, type andfrequency of the eye tracker or a global parameter for determiningexpected gaze point deviation which may override other factors. If thedesired element can be reliably determined, the computer 100 canoptionally highlight the element at the non-contact target at block2908. The computer 100 can then perform a computer function associatedwith the element at the non-contact target (e.g., selecting the elementor opening the element or others). Secondly, the computer 100 determinespossible targets at block 2912 based on factors including geometries ofelements near the gaze target such as how close together multipleelements are and the size and/or shape of the elements, layout of theelements, visual point of gravity of the elements representingweightings of the elements such that a gaze target may be weightedtowards an element in a gravity-like manner, grouping criteria includinggrouping elements of the same interaction type, contextual input fromthe user such as a preference towards avoiding functions such as deleteor the like and whether based on the user's gaze point the user has seenan element highlight. At block 2914 the computer 100 determines theproperties of a region in which to present a visualization of allpotential elements. The computer 100 may consider various factors suchas the size and layout of elements including possible targets determinedat block 2912, a maximum size of a visualization of spaced apartelements, a magnification level set by the user or predetermined by thecomputer 100 and grouping criteria for analyzing displayed elements. Thecomputer 100, at block 2914, can then present a visualization of allpotential elements identified in block 2912 with each element spacedfurther apart. Optionally, elements displayed at block 2914 can behighlighted as a user's gaze locates upon them. An example of spacingfurther apart at block 2914 can include simply zooming in on the areawhere the elements are located (i.e., increasing the space between theelements with respect to the size of the display device 104). Anotherexample of spacing further apart at block 2914 can include enlargingand/or moving each individual element further apart on the display toprovide greater separation (e.g., moving four elements in a squareconfiguration taking up a small portion of the display to a lineformation extending across substantially most of the display). At blocks2916 and 2918, the computer 100 can determine the non-contact targetagain and detect a second action, respectively, in order to identify atargeted element. The computer 100 can optionally highlight the targetedelement at block 2908. The computer 100 can perform a computer functionassociated with the targeted element at block 2910 (e.g., clicking theelement).

In enhancements to the embodiment shown in FIG. 29, at block 2904 thecomputer 100 can receive a maintained first action, such as constantpressure on a touchpad, a held down button, or the like. As the user ismaintaining a first action, the user is actively selecting elements andthus is prepared for multiple elements to be highlighted prior to block2906 and at block 2914. At block 2910 the user releases the maintainedfirst action and the computer 100 performs a computer functionassociated with the element at the non-contact target.

Further Embodiments

In some embodiments, the computer 100 is able to detect non-contactactions to switch the focus of windows in the display. In an embodiment,when a user looks at a non-active window and initiates an action (e.g.,a contact-required action or a non-contact action), the non-activewindow becomes the active window. In this embodiment, an eye trackingdevice determines the location of a user's gaze relative to a display200, if the computer 100 determines the gaze target to be located in awindow or area not currently in an active state, the user can instructthe computer 100 to make the window or area active by providing acontact-required action whilst gazing at the non-active window or area.In alternate embodiments, the user can fixate their gaze within thenon-active window or area for a predetermined period of time instead ofproviding a contact-required action, in order to make the window or areaactive.

In an embodiment, a user can scroll a window or screen area by lookingtowards an edge of the screen, window, or area and initiating an actionsuch as a contact-required action. Depending on whether the user waslooking towards the top, bottom, left, right, or diagonal edge of thescreen, window, or area, the area or window can scroll up, down, left,right or diagonal. For example, if a user is looking towards the bottomof an area or window and initiates an action such as a contact-requiredaction, the contents of the area or window at the bottom of the area orwindow can scroll upwards to the top of the window or area, effectivelyrevealing new information from the bottom of the area or window. Thisfunctionality can operate in the same manner for every direction atwhich the user is gazing. For example if the user is gazing towards theleft edge when initiating an action, the information at the left edgewill move towards the right edge and effectively scroll information fromof the left of the area or window towards the right of the area orwindow. In an embodiment, when a user looks at a point within a windowor area and initiates an action (e.g., a contact-required action), thecomputer 100 will scroll the window or area so that point identified bythe user is moved to a predetermined location in the window (e.g., thetop of the window or the center of the window). In these embodiments,where an area such as an edge, top/bottom, left/right, or other ismentioned, the computer 100 can determine based on a gaze offset ordeviation an area near the edge, top/bottom, left/right, or other inwhich to enact the functions described.

In some embodiments, the computer 100 is able to directly scroll thewindow by a predetermined amount for gaze interactions. In alternateembodiments, the computer 100 uses the gaze interactions to emulatebutton presses, such as presses of arrow buttons (e.g., arrow up orarrow down) or page buttons (e.g., page up or page down), in order toscroll the window.

In some embodiments, the computer 100 can provide a visual indication(e.g., a different cursor or other indication) that the gaze target islocated in a particular zone or area defined for scrolling (e.g., nearthe top edge of the screen for scrolling up).

In an embodiment, a user can scroll a window by holding acontact-required input (e.g., a button or touchpad) while gazing in adirection of scroll (e.g., top, bottom, or side of the computer screen).A gaze to an edge of a screen, window, or area can scroll informationfrom that edge towards the center of the screen, window, or area. Thescrolling can be momentum based scrolling, meaning that the rate ofscrolling will gradually increase to a predetermined maximum speed thelonger the contact-required input is held (e.g., the longer the buttonis pressed or the longer the touchpad is touched without release). Insome embodiments, once the button or touchpad is released, the scrollingwill not cease immediately, but will rather slow rapidly until coming toa complete stop. The speed at which the scrolling increases whilst thebutton or touchpad is held down and at which the scrolling decreasesafter release can be adjusted by the computer 100. Optionally theadjustment can be altered by a user.

In some embodiments, a user can scroll a window simply by looking at theedge of the window (e.g., the top, bottom, or sides of the window). Thewindow can scroll by a predetermined increment for each look towards theedge of the window. In some embodiments, the scrolling will only happenif the user looks towards the edge of the window and simultaneouslyinitiates an action, such as a non-contact action like a voice command.In other embodiments, the increment can depend on the location of theuser's gaze and can update continuously. In this way natural scrollingis achieved, for example if a user gazes towards the edge of a mapconstantly the map scrolls by small increments to smoothly scroll themap. The scrolling can occur without the user simultaneously initiatingan action.

In some embodiments, a user can scroll a window by looking at a desiredlocation and performing an action (e.g. pressing a button or saying“scroll”), at which point the computer 100 will cause the portion of thewindow and/or display located at the gaze target to move to the centerof the window and/or display. In some embodiments, a computer 100 candetermine whether to scroll or zoom, depending upon the location of thegaze target (e.g., location of the gaze target with respect to awindow). For example, if a user looks at a window containing a map andmoves the gaze target to an edge of the map, pressing a button can causethe map to scroll so the targeted location is now at the center of themap. However, if the same user looks at or near the center of the map,pressing a button can cause the map to zoom. In some additionalembodiments, the speed of an action (e.g., zooming or scrolling) can becontrolled based on the location of the gaze target (e.g., location ofthe gaze target with respect to a window). For example, when a userlooks at the edge of a window and presses a button, the window canscroll quickly, whereas if the user looks somewhere between the sameedge of the window and the center of the window, the window would scrollless quickly.

In some embodiments, the scrolling is based on the length or pressure ofcontact by a user upon a key, button, or touchpad. For example, a windowcan scroll by greater increments when a user holds a key, button, ortouchpad down longer.

In various embodiments, scrolling can be terminated by performing orceasing actions (e.g., contact-required actions). In some embodiments,scrolling is slowed before completely terminating. In some embodiments,scrolling can be slowed by the user gradually moving the gaze targetaway from a predetermined area (e.g., edge of the screen or area wherethe gaze target was upon initiation of scrolling).

In some embodiments, a tap of a contact-required input (e.g., quickbutton press, quick tap on a touchpad, or movement of a computer mouse)can cause a visual indicator to appear on the display, or alternativelyenact a click at the last known position of a visual indicator or at thegaze location. Thereafter, holding the finger on the contact-requiredinput (e.g., holding a finger on a touchpad) can move the visualindicator to the gaze target.

In some embodiments, movement of a computer mouse can cause the visualindicator to appear on the display at the gaze target. In someembodiments, after the visual indicator has appeared at the gaze target,use of a contact-required input (e.g., computer mouse) to move thevisual indicator (e.g., cursor) can be slowed while the visual indicatoris near the user's gaze target. In some embodiments, a user can initiatean additional action (e.g., a contact-required action such as pressingand holding a mouse button) on an element (e.g., an icon) while using acomputer mouse. The user's gaze can then be moved towards a desireddestination and the mouse button can be released in order to drag theelement to the desired destination (e.g., to move an item into anotherfolder). In this manner, a user can select an item on a display bylocating a visual indicator on or near the item either by using a gazeenabled visual indicator movement method described previously, or bymoving a computer mouse, touchpad or the like. The user can then holddown an activator such as a computer mouse or maintain contact on atouchpad to select the item, when the user releases the activator, theitem moves to the location of the user's gaze on the display. Therefore,the user can “grab” an item such as an icon and “drag” the icon to theuser's gaze location, whereupon on release of an activator the icon isrelocated to the user's gaze location.

In some embodiments, performing an action such as a contact-requiredaction (e.g., clicking a mouse button) can cause the visual indicator tomove to the gaze target. In some embodiments, holding thecontact-required action (e.g., pressing and holding the mouse button)can move the visual indicator to the gaze target and allow for the userto fine-tune the position of the visual indicator before releasing thecontact-required action (e.g., releasing the mouse button) to perform anaction at the visual indicator's location (e.g., select an element).While the contact-required action is being held, large movements of gazetarget can translate to smaller movements of the visual indicator, inorder to allow a user to more finely adjust the location of the visualindicator.

In an embodiment, initiating an action (e.g., a contact-required action)can open a menu at the gaze target. A user can look at the desired menuitem and initiate a second action (e.g., second contact-required actionor release of a maintained first contact-required action) to select thatmenu item. In some embodiments, holding gaze on or near the menu itemfor a predetermined period of time can act to select that menu item.

In an embodiment, an edge menu or button can be opened (e.g., displayed)by holding a contact-required action, looking to the edge of thedisplay, and releasing the contact-required action. In some cases, thecomputer 100 presents a visual indication that opening a menu ispossible when the user looks to the edge of the screen with thecontact-required action held. If the user looks away from the edge ofthe display without releasing the contact-required action, the menu doesnot open. For example the user can hold or maintain a contact-requiredaction and look towards an edge of a display. A glow or other visualindicator can appear at the edge of the display indicating a menu can beopened. If the contact-required action is ceased while the glow or othervisual indicator is present, the menu appears on screen. If thecontact-required action is ceased while the glow or other visualindicator is not present, no menu appears on screen. The menu can occupythe full screen or part of the screen. The menu can alternatively be abutton, for example a button representing a “back” movement in a webbrowser the like.

In an embodiment, initiating an action (e.g., a contact-required actionsuch as actuating a scroll wheel) will zoom the display in or out at thegaze target. In an embodiment, initiating an action (e.g., a double-tapon a touchpad) can zoom the display in at the gaze target. In anembodiment, initiating an action (e.g., tapping the touchpad whileholding down a shift button) can zoom the display out at the gazetarget. In a further embodiment, movement of two fingers towards eachother such as a “pinch” movement on a touchpad, touch screen or the likecan enact a zoom in or out, and movement of two fingers away from eachother can enact the opposite zoom in or out movement.

In an embodiment, in order to activate a non-contact input mode, theuser looks towards the edge of the display device 104. The computer 100determines that the gaze target is at or near the edge of the displaydevice 104 and activates a non-contact input mode. In some embodiments,the computer 100 displays an input menu over or adjacent the display200. When the user looks away from the edge of the display 200, theinput menu can disappear immediately, remain on the display 200indefinitely, or disappear after a predetermined amount of time. Theuser can activate an icon on the input menu by performing acontact-required action while gazing at an icon or by using anon-contact activation method such as dwelling their gaze in thevicinity of an icon for a predetermined period of time, for example onesecond, or blinking an eye or eyes, which can be interpreted by thecomputer as an activation command. Each icon can be associated with acomputer function. When an icon is activated, the computer 100 canprovide an indication of activation, such as a change in the appearanceof the icon, a sound, physical feedback (e.g., a haptic response), orother indication.

A place cursor icon can be activated to place the mouse cursor on adesired point or position. The place cursor icon can be used formouse-over functions (e.g., functions where a mouse click is notdesired). A gaze scroll icon can be activated to enable gaze-controlledscrolling within a scrollable window, as described in further detailbelow. A left click icon can be activated to perform a single left-click(e.g., emulate a physical left-click on an attached computer mouse). Adouble click icon can be activated to perform a double left-click (e.g.,emulate a physical double-click on an attached computer mouse). A rightclick icon can be activated to perform a single right-click (e.g.,emulate a physical right-click on an attached computer mouse). A gazedrag and drop icon can be activated to enable the gaze drag and dropmode. The gaze drag and drop mode allows a user to use non-contact inputto emulate a drag and drop action on a physical mouse (e.g., click andhold the mouse, move the mouse, release the mouse), as described infurther detail below. A gaze keyboard icon can be activated to open anon-screen, gaze-enabled keyboard for typing using gaze, as described infurther detail below. A settings icon can be activated to open asettings window or dialog.

As described above, the gaze scroll icon can be activated to enablegaze-controlled scrolling. When gaze-controlled scrolling is activated,the user can scroll windows (e.g., up and down, as well as left andright) using non-contact inputs. In one embodiment, the user can place ascroll indicator on the window and look above the scroll indicator toscroll up, look below the scroll indicator to scroll down, look to theleft of the scroll indicator to scroll left, and look to the right ofthe scroll indicator to scroll right. The user can place the scrollindicator by first enabling gaze-controlled scrolling (e.g., dwell gazeon the gaze scroll icon), then looking at any scrollable area anddwelling gaze upon that area until the scroll indicator appears. Whenthe user desires to disable gaze-controlled scrolling, the user can gazeoutside of the screen or back to the input menu.

As described above, the gaze drag and drop icon can be activated toenable the gaze drag and drop mode. When the gaze drag and drop mode isenabled, the user can gaze at a first location and provide a user signal(e.g., dwelling gaze, blinking, winking, blinking in a pattern, or usinga contact input such as a button) which causes the computer to emulate amouse click and hold at the first location. The user can then move gazeto a second location and provide a second user signal, which causes thecomputer to emulate moving the mouse to the second position andreleasing the mouse button.

In some embodiments, once a user has selected an icon on the input menu,the icon selected may not be de-selected unless a new selection on theinput menu is made. In other embodiments, the icon can be de-selected,such as by gazing at the same icon again.

In some embodiments, to facilitate using the action selected on theinput menu, the computer 100 can provide for a portion of the display tobe zoomed (e.g., displayed at a lower resolution). For example, when theuser selects the “left click” icon on the input menu and gazes at aportion of the display, an area around the gaze point on the display canzoom so that the user can then select the intended target for theiraction with greater ease by gazing at the enlarged portion of thedisplay.

In some cases, the user can perform a certain action (e.g.,contact-required action or non-contact action) in order to select anarea of the display upon which a computer function is to be performed,at which point the computer 100 can display an input menu at or aroundthat point to select the desired computer function.

In a further improvement, the input menu can also be provided externalto the display 200 and/or display device 104. In this manner it can beprovided on an input device such as an eye tracking device, on thehousing of a display, or on a separate device. The menu can then becomprised of a separate display, or another means of conveyinginformation to a user such as light emitting diodes, switches or thelike. In another embodiment, the action of choosing an icon on theexternal input menu is shown as a transparent image of that icon at anappropriate position on the regular display. In some embodiments, thegaze target used to identify the desired icon to be activated can belocated outside the display 200 and/or display device 104.

In some embodiments, a user can perform computer functions on or at thearea at or near the gaze target via voice interaction. Once an area hasbeen selected (e.g., by focusing gaze at the area), an action can beperformed by the user speaking certain words such as “open”, “click” andthe like, which would be readily understood by a skilled addressee.

In some embodiments, selection of a computer function on an input menucan comprise multiple steps and can also comprise multiple menus. Forexample a user can select an icon or menu using any input method hereindescribed, and then select a second icon or menu for selecting acomputer function to be performed.

In some embodiments, a gaze-tracking device can discern the identity ofthe user (e.g., through a user's gaze patterns or biometric data such asdistance between eyes and iris sizes) in order to customizefunctionality for that particular user (e.g., use particular menusand/or features or set up desired brightness settings or othersettings).

In some embodiments, a computer 100 will perform certain functions basedon a discernible series or sequence of gaze movements (e.g., movementsof the gaze target).

In some embodiments, a computer 100 can determine what various computerfunctions are available to be performed in whole or in part bynon-contact actions based on what elements are presented on the display200.

In some embodiments, performing an action can cause information to bepresented on the display 200 based on the gaze target. For example,pressing a button can cause a list of active computer programs to appearon the display 200. A user can interact with the presented information(e.g., list of active computer programs) by gazing at parts of theinformation. For example, a user can press a button to cause a list ofactive computer programs to appear on the display 200, then look at aparticular program and release the button in order to cause thatparticular program to gain focus on the computer 100.

Many examples presented herein are described with respect to gazetracking. It will be understood that, where applicable, tracking ofother non-contact actions (e.g., 3-D gestures or others) can besubstituted for gaze tracking. When tracking of other non-contactactions is substituted for gaze tracking, references to gaze targetsshould be considered references to non-contact targets.

Location of a gaze target can be determined from detection of variousactions, including movement of the eyes, movements of the face, andmovements of facial features. For example, scrolling function (e.g.,scrolling a page up or down) can be controlled by the user's facetilting (e.g., up or down) while the user reads the display 200, whereinthe computer 100 does not control scrolling function based on eyetracking at that time.

In some cases, camera based gaze detection systems can rely on facialrecognition processing to detect facial features such as nose, mouth,distance between the two eyes, head pose, chin etc. Combinations ofthese facial features can be used to determine the gaze target. Forexample, in embodiments where vertical scrolling (e.g., a scroll upfunction and/or a scroll down function) is to be performed based on faceimages from a camera, the detection of the gaze target can rely solelyor in part on detected eyelid position(s). When the user gazes at thelower portion of the display device 104, the eye will be detected asbeing more closed, whereas when the user gazes at the top of the displaydevice 104, the eye will be detect as being more open.

Eye lid position detection is good for determining changes in gazetarget in a vertical direction, but not as effective for determiningchanges in gaze target in a horizontal direction. For better determiningchanges in gaze target in a horizontal direction, images of the headpose can be used instead. In such cases, a gaze target can be determinedto be within scroll zones only when the user's face is determined to beoriented in the general direction of the display device 104. As generalrule, whenever a user looks at an object more than seven degrees offfrom a direct forward line of sight, the user's head will immediatelyturn in the direction of that object. Thus, a head pose indicating morethan seven degrees off to a side from the display device 104 is anindication that the user is unlikely to be looking at the content (e.g.,the display 200) presented on the display device 104.

Depending on the sensitivity and accuracy of the gaze detectioncomponents, which can be dictated by camera resolution, processingpower, available memory, and the like, a gaze target can occupy asmaller (e.g., more sensitive and/or accurate) or larger (e.g., lesssensitive and/or accurate) area relative to the display device 104.Calibration of the gaze detection components can also play a role in theaccuracy and sensitivity of gaze target calculations. Accuracy orsensitivity can dictate the relationship between an actual direction ofa user's gaze and the calculated gaze target. The disclosed embodimentscan function even if the relationship between the actual gaze directionand calculated gaze target is not direct.

In some embodiments, the gaze target can be calibrated by using inputfrom a touch screen to assist with calibration. For example, thecomputer 100 can prompt the user to look at and touch the same point(s)on the display device 104. Alternatively, such a calibration process canbe performed in the background without prompting the user orinterrupting the user's normal interaction with the computer 100. Forexample, while normally operating the computer 100, a user will bepressing buttons, hyperlinks, and other portions of the display 200,display device 104, and/or computer 100 having known positions. It canbe assumed that the user will normally also be looking at the buttons,hyperlinks, etc. at the same time. Thus, the computer 100 can recognizethe touch point or click point as direction of the user's gaze and thencorrect any discrepancies between the direction of the user's gaze andthe calculated gaze target. Such a background calibration process can behelpful in order to slowly improve calibration as the user interactswith the computer 100 over time.

In some embodiments, a computer 100 is able to determine when a user isreading elements 504 on a display 200 rather than attempting to controlthe computer 100. For example, detection of whether a user is readingcan be based on detecting and evaluating saccades and whether an eyefixates or dwells on or around a constant point on the display. Thisinformation can be used to determine indicators of reading asdistinguished from a more fixed gaze. In some embodiments, the computer100 is configured such that scrolling functions can be initiated evenwhen the user is determined to be reading. For instance when the user islooking at a map, the scrolling (e.g., panning) should be initiatedrelatively faster than when the user is reading text (e.g., aword-processor document). Thus any dwell time before triggering a scrollfunction when reading text can be longer than for reviewing maps andother graphical content, and the scroll zones and/or scroll interactionscan be chosen differently in each case. For example, the scroll zone(s)may have to be made larger in the case of a map or other graphicalcontent in order to make the computer 100 sufficiently responsive, whilescroll zone(s) for a text document may be smaller because a scrollaction is typically not required until the user is reading text veryclose (e.g., 5 lines) to the bottom or top of a window.

Various embodiments are disclosed herein that can correlate certainactions with computer functions. In additional embodiments, whereapplicable, a computer 100 can be placed in one or more modes, whereineach mode enables different computer functions to be performed inresponse to a user performing various actions.

In some embodiments, the computer 100 can be configured to use gaze datapatterns (e.g., the frequency with which gaze targets appear in certainpositions or locations relative the display device 104) to determinewith greater accuracy, based on statistical analysis, when a user islooking at a particular zone 202 or element 504.

FIGS. 8-24D depict graphical representations and flow charts of variousembodiments and functionality disclosed herein.

Although embodiments have been described referencing contact requiredand non-contact required actions, it is intended to be understood thatthese actions can be interlaced. In other words, if an embodiment isdescribed using a contact required action such as movement of a mouse,touchpad contact, pressing of a button or the like, it is intended thatsuch an action can also be performed by using a non-contact method suchas a voice command, gesture, gaze movement or the like.

All patents, publications and abstracts cited above are incorporatedherein by reference in their entirety. Any headers used herein are fororganizational purposes only and are not to be construed to limit thedisclosure or claims in any way. Various embodiments have beendescribed. It should be recognized that these embodiments are merelyillustrative of the principles of the present disclosure. Numerousmodifications and adaptations thereof will be readily apparent to thoseskilled in the art without departing from the spirit and scope of thepresent disclosure as defined in the following claims.

Unless specifically stated otherwise, it is appreciated that throughoutthis specification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining,” and “identifying” or the likerefer to actions or processes of a computing device, such as one or morecomputers or a similar electronic computing device or devices, thatmanipulate or transform data represented as physical electronic ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of thecomputing platform.

The system or systems discussed herein are not limited to any particularhardware architecture or configuration. A computing device can includeany suitable arrangement of components that provides a resultconditioned on one or more inputs. Suitable computing devices includemultipurpose microprocessor-based computer systems accessing storedsoftware that programs or configures the computing system from a generalpurpose computing apparatus to a specialized computing apparatusimplementing one or more embodiments of the present subject matter. Anysuitable programming, scripting, or other type of language orcombinations of languages may be used to implement the teachingscontained herein in software to be used in programming or configuring acomputing device.

Embodiments of the methods disclosed herein may be performed in theoperation of such computing devices. The order of the blocks presentedin the examples above can be varied—for example, blocks can bere-ordered, combined, and/or broken into sub-blocks. Certain blocks orprocesses can be performed in parallel.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited.

As used below, any reference to a series of examples is to be understoodas a reference to each of those examples disjunctively (e.g., “Examples1-4” is to be understood as “Examples 1, 2, 3, or 4”).

Example 1 is a method of controlling a computer. The method includespresenting a display having a visual indicator; detecting a gaze targetof a user; detecting a contact-required action of a user; and performinga computer function in response to the contact-required action and thegaze target. Performing the computer function includes performing afirst function. The first function can be scrolling a first portion ofthe display, in response to detection of the contact-required action,based on the location of the gaze target with respect to the display.The first function can be scrolling a second portion of the display, inresponse to continued detection of the contact-required action, based onthe location of the gaze target with respect to the display. The firstfunction can be moving the visual indicator to the gaze target inresponse to the contact-required action. The first function can bemoving the visual indicator at a first rate slower than a second rate ofmovement of the gaze target in response to continued detection of thecontact-required action. The first function can be zooming a thirdportion of the display adjacent the gaze target in response to thecontact-required action. The first function can be performing a secondfunction, in response to the contact-required action, based on the gazetarget when the gaze target is outside the display. The first functioncan be performing a second computer function, in response to thecontact-required action, based on a sequence of movements of the gazetarget.

Example 2 is the method of example 1, where the first function isscrolling the second portion of the display, in response to continueddetection of the contact-required action, based on the location of thegaze target with respect to the display. The continued detection of thecontact-required action is detection of continued touching of atouchpad.

Example 3 is the method of examples 1 or 2, where the first functionadditionally includes increasing the momentum of the scrolling untildetection of continued touching of the touchpad ceases.

Example 4 is the method of example 1 where the first function is movingthe visual indicator to the gaze target in response to thecontact-required action. The contact-required action is one of the groupconsisting of touching a touchpad or moving a computer mouse.

Example 5 is the method of example 1, where the first function iszooming a third portion of the display adjacent the gaze target inresponse to the contact-required action. The contact-required action isactuation of a scroll wheel.

Example 6 is the method of example 1, where the first function iszooming a third portion of the display adjacent the gaze target inresponse to the contact-required action. The contact-required action isa combination of depressing a button and performing a secondcontact-required action.

Example 7 is the method of example 1, where the contact-required actionis touching a touch-sensitive device.

Example 8 is a computing device having a computer including a displaydevice for presenting a display having a visual indicator. The computerfurther includes a contact-required input and a non-contact input. Thecomputer is programmed to detect a non-contact target of a user anddetect a contact-required action of the user. The computer is furtherprogrammed to perform a computer function, in response to thecontact-required action, based on the non-contact target, whereinperforming the computer function includes performing a first function.The first function can be scrolling a first portion of the display, inresponse the contact-required action, based on the location of thenon-contact target with respect to the display. The first function canbe scrolling a second portion of the display, in response to continueddetection of the contact-required action, based on the location of thenon-contact target with respect to the display. The first function canbe moving the visual indicator to the non-contact target in response tothe contact-required action. The first function can be moving the visualindicator at a first rate slower than a second rate of movement of thenon-contact target in response to continued detection of thecontact-required action. The first function can be zooming a thirdportion of the display adjacent the non-contact target in response tothe contact-required action. The first function can be performing asecond function, in response to the contact-required action, based onthe non-contact target when the non-contact target is outside thedisplay. The first function can be performing a second computerfunction, in response to the contact-required action, based on asequence of movements of the non-contact target.

Example 9 is the computing device of example 8, where the first functionis scrolling the second portion of the display, in response to continueddetection of the contact-required action, based on the location of thenon-contact target with respect to the display. The continued detectionof the contact-required action is detection of continued touching of atouchpad.

Example 10 is the computing device of example 9, where the firstfunction additionally includes increasing the momentum of the scrollinguntil detection of continued touching of the touchpad ceases.

Example 11 is the computing device of example 8, where the firstfunction is moving the visual indicator to the non-contact target inresponse to the contact-required action. The contact-required action isone of the group consisting of touching a touchpad or moving a computermouse.

Example 12 is the computing device of example 8, where the firstfunction is zooming a third portion of the display adjacent thenon-contact target in response to the contact-required action. Thecontact-required action is actuation of a scroll wheel.

Example 13 is the computing device of example 8, where the firstfunction is zooming a third portion of the display adjacent thenon-contact target in response to the contact-required action. Thecontact-required action is a combination of depressing a button andperforming a second contact-required action.

Example 14 is the computing device of example 8, where thecontact-required action is touching a touchpad.

Example 15 is the computing device of example 8, where the non-contactinput is a gaze-tracking device and the non-contact target is a gazetarget.

Example 16 is a system having a computer, the computer including adisplay device for presenting a display having a visual indicator; aneye-tracking device for detecting a gaze target; a contact-requiredinput for detecting a contact-required action; and a processor operablyconnected to the eye-tracking device, contact-required input, anddisplay device. The computer further includes programming enabling theprocessor to perform a computer function, in response to thecontact-required action, based on the non-contact target, whereinperforming the computer function includes performing a first function.The first function can be scrolling a first portion of the display, inresponse the contact-required action, based on the location of the gazetarget with respect to the display. The first function can be scrollinga second portion of the display, in response to continued detection ofthe contact-required action, based on the location of the gaze targetwith respect to the display. The first function can be moving the visualindicator to the gaze target in response to the contact-required action.The first function can be moving the visual indicator at a first rateslower than a second rate of movement of the gaze target in response tocontinued detection of the contact-required action. The first functioncan be zooming a third portion of the display adjacent the gaze targetin response to the contact-required action. The first function can beperforming a second function, in response to the contact-requiredaction, based on the gaze target when the gaze target is outside thedisplay. The first function can be performing a second computerfunction, in response to the contact-required action, based on asequence of movements of the gaze target.

Example 17 is the system of example 16 where the first function isscrolling the second portion of the display, in response to continueddetection of the contact-required action, based on the location of thegaze target with respect to the display. The continued detection of thecontact-required action is detection of continued touching of atouchpad.

Example 18 is the system of example 16, where the first functionadditionally includes increasing the momentum of the scrolling untildetection of continued touching of the touchpad ceases.

Example 19 is the system of example 16, where the first function ismoving the visual indicator to the gaze target in response to thecontact-required action. The contact-required action is one of the groupconsisting of touching a touchpad or moving a computer mouse.

Example 20 is the system of example 16, where the first function iszooming a third portion of the display adjacent the gaze target inresponse to the contact-required action. The contact-required action canbe actuation of a scroll wheel. The contact-required action can be acombination of depressing a button and performing a secondcontact-required action.

Example 21 is the system of example 16, where the contact-requiredaction is touching a touchpad.

In another embodiment of the invention, a method for providing input toa user interface using a gaze interaction is provided. The method mayinclude determining a gaze point of a user on a user interface hasremained on a particular area (for example, a text entry field) of theuser interface for a first predetermined amount of time, where theparticular area of the user interface is configured to accept a textstring input. Upon receiving an input from a non-gaze input device, andbased upon the gaze point of the user remaining in the particular areafor the first predetermined amount of time prior to receiving the inputfrom the non-gaze input device, the input is then provided from thenon-gaze input device to the particular area of the user interface.

In this manner, a method is provided which allows for quick text entryinto fields of an interface without use of a mouse or other pointingdevice or means (i.e., touch screen, trackball, etc.) to first selectthe text field as the new input focus. In many typical graphical userinterfaces, such as Windows™ or OS X™, input focus is defined as thecurrently active graphical user interface element in which a keystrokefrom the keyboard will be delivered. For example, while a user may havefive windows or applications open, only one at a time will have theinput focus, and be capable of receiving keyboard entry.

Thus, in embodiments of the invention, a user may be using anyparticular portion of a user interface, and then direct their gaze to atext field on the user interface intending to input text into the field,regardless of whether it currently has input focus. Upon passing of afirst predetermined amount of time (e.g., any amount of time in 1millisecond increments between zero and five seconds), the method maypredict that a text input from a keyboard, touch display keyboard, orother input device is intended for the text field. In some embodiments,upon passing of the predetermined amount of time, the text field may behighlighted to indicate to the user that the method believes a textinput from the keyboard, touch display keyboard, or other input deviceis intended for the text field. Upon actual input of text via thenon-gaze input device, input focus may be changed.

In some embodiments, the gaze point of the user, while actuallysignaling an intent to deliver text entry to the gazed at text field,may not consistently stay in the text field, but may lie away from thetext field for small amounts of time during the predetermined amount oftime. In various embodiments, a limit (or second predetermined amount oftime) may be established whereby the gaze point leaving the text fieldfor time periods of less than the limit will be ignored for purposes ofdetermining if the user's gaze point has remained in the text field forat least the first predetermined amount of time. The secondpredetermined amount of time may be any amount of time in 1 millisecondincrements between zero and five seconds.

In some embodiments, certain directional movements of the gaze pointaway from the text field during the predetermined amount of time canalso be ignored by the method. For example, downward movements of thegaze point away from the text field may indicate the user is looking ata non-gaze input device (i.e., keyboard, touch display keyboard, orother input device) in order to correctly type or otherwise input a textstring. In some embodiments, such as touch display keyboards, the methodmay be able to know whether the user's gaze point has moved to suchnon-gaze input devices. In other embodiments, such as computers usingphysical keyboards, the method may be able to infer from movement of thegaze point off the bottom of the display that the user is looking at thekeyboard. In any of the above cases, the method may ignore suchmovements of the user's gaze point away from the text field.

The above described methods of changing input may be particularlyadvantageous for user interfaces in which there are a limited number of,or a single important, text entry field. Merely by way of example, suchuser interfaces may include a web browser's Uniform Resource Locater(URL) field, an operating systems search field (e.g., Windows 10™Cortan™ a search assistant), a messaging interface's send field (e.g.,AOL™ Instant Messenger™), and/or a notification's response field (e.g.,Skype™ toast notification messaging field).

Turning now to FIG. 30, a method 3000 of the invention for providinginput to a user interface using a gaze interaction is shown in blockdiagram format. At block 3010, a user's gaze point is determined. Atblock 3020, it is determined that the user's gaze point is resting on aparticular area. At block 3030, it is determined whether or not the gazepoint has rested on the particular area for at least a firstpredetermined amount of time. If so, at block 3040, an input at anon-gaze input device is awaited. Upon receipt, at block 3050 inputfocus is changed in the user interface to the particular area, typicallya text entry field. At block 3060 input received from the non-gaze inputdevice is provided to the particular area as input.

At block 3030, if it is determined that the user's gaze point leaves theparticular area, at block 3070 it is determined if the time that theuser's gaze point is away from the particular area is less than a secondpredetermined amount of time, or if the gaze point has moved in apredefined direction which indicates a likely text input (such as akeyboard beneath the display, etc.). If so, method 3000 continues atblock 3040. If not, method 3000 begins again at block 3010.

What is claimed is:
 1. A method for providing input to a user interfaceusing a gaze interaction, wherein the method comprises: determining agaze point of a user on a user interface has remained on a particulararea of the user interface for a first predetermined amount of time,wherein the particular area of the user interface is configured toaccept a text string input; and receiving an input from a non-gaze inputdevice, and based upon the gaze point of the user remaining in theparticular area for the first predetermined amount of time prior toreceiving the input from the non-gaze input device, providing the inputfrom the non-gaze input device to the particular area of the userinterface.
 2. The method for providing input to a user interface using agaze interaction of claim 1, wherein the method further comprises:changing an input focus from a different portion of the interface to theparticular area upon receiving the input from the non-gaze input device.3. The method for providing input to a user interface using a gazeinteraction of claim 1, wherein determining the gaze point of the userhas remained on the particular area for the first predetermined amountof time comprises: ignoring the gaze point of the user leaving theparticular area for a second predetermined amount of time during thefirst predetermined amount of time.
 4. The method for providing input toa user interface using a gaze interaction of claim 1, wherein the methodfurther comprises: determining that the gaze point of the user has movedto a non-gaze input device after the first predetermined amount of time,and continuing to provide the input from the non-gaze input device tothe particular area of the user interface after the first predeterminedamount of time.
 5. The method for providing input to a user interfaceusing a gaze interaction of claim 4, wherein determining that the gazepoint of the user has moved to a non-gaze input device comprises:determining that the gaze point of the user has moved to below the userinterface or a display on which the user interface is presented.
 5. Themethod for providing input to a user interface using a gaze interactionof claim 4, wherein the non-gaze input device comprises: a keyboardbelow the user interface.
 6. The method for providing input to a userinterface using a gaze interaction of claim 4, wherein the non-gazeinput device comprises: a touch display keyboard below the userinterface.
 7. The method for providing input to a user interface using agaze interaction of claim 1, wherein: the user interface comprises a webbrowser; and the particular area comprises a Uniform Resource Locaterfield.
 8. The method for providing input to a user interface using agaze interaction of claim 1, wherein: the user interface comprises agraphical user interface of an operating system; and the particular areacomprises a search field.
 9. The method for providing input to a userinterface using a gaze interaction of claim 1, wherein: the userinterface comprises a messaging interface; and the particular areacomprises a send field.
 10. The method for providing input to a userinterface using a gaze interaction of claim 1, wherein: the userinterface comprises a notification; and the particular area comprises aresponse field.
 11. A non-transitory machine readable medium havinginstructions stored thereon for providing input to a user interfaceusing a gaze interaction, wherein the instructions are executable by oneor more processors for at least: determining a gaze point of a user on auser interface has remained on a particular area of the user interfacefor a first predetermined amount of time, wherein the particular area ofthe user interface is configured to accept a text string input; andreceiving an input from a non-gaze input device, and based upon the gazepoint of the user remaining in the particular area for the firstpredetermined amount of time prior to receiving the input from thenon-gaze input device, providing the input from the non-gaze inputdevice to the particular area of the user interface.
 12. Thenon-transitory machine readable medium of claim 11, wherein theinstructions are further executable for at least: changing an inputfocus from a different portion of the interface to the particular areaupon receiving the input from the non-gaze input device.
 13. Thenon-transitory machine readable medium of claim 11, wherein determiningthe gaze point of the user has remained on the particular area for thefirst predetermined amount of time comprises: ignoring the gaze point ofthe user leaving the particular area for a second predetermined amountof time during the first predetermined amount of time.
 14. Thenon-transitory machine readable medium of claim 11, wherein theinstructions are further executable for at least: determining that thegaze point of the user has moved to a non-gaze input device after thefirst predetermined amount of time, and continuing to provide the inputfrom the non-gaze input device to the particular area of the userinterface after the first predetermined amount of time.
 15. Thenon-transitory machine readable medium of claim 14, wherein determiningthat the gaze point of the user has moved to a non-gaze input devicecomprises: determining that the gaze point of the user has moved tobelow the user interface or a display on which the user interface ispresented.
 16. A system for providing input to a user interface using agaze interaction, wherein the system comprises: an eye tracking devicefor determining a gaze point of a user on a user interface; at least oneprocessor for at least: determining the gaze point of the user on theuser interface has remained on a particular area of the user interfacefor a first predetermined amount of time, wherein the particular area ofthe user interface is configured to accept a text string input; anddetermining that an input from a non-gaze input device has beenreceived, and based upon the gaze point of the user remaining in theparticular area for the first predetermined amount of time prior toreceiving the input from the non-gaze input device, providing the inputfrom the non-gaze input device to the particular area of the userinterface.
 17. The system for providing input to a user interface usinga gaze interaction of claim 16, wherein the at least one processor isfurther for at least: changing an input focus from a different portionof the interface to the particular area upon receiving the input fromthe non-gaze input device.
 18. The system for providing input to a userinterface using a gaze interaction of claim 16, wherein determining thegaze point of the user has remained on the particular area for the firstpredetermined amount of time comprises: ignoring the gaze point of theuser leaving the particular area for a second predetermined amount oftime during the first predetermined amount of time.
 19. The system forproviding input to a user interface using a gaze interaction of claim16, wherein the at least one processor is further for at least:determining that the gaze point of the user has moved to a non-gazeinput device after the first predetermined amount of time, andcontinuing to provide the input from the non-gaze input device to theparticular area of the user interface after the first predeterminedamount of time.
 20. The system for providing input to a user interfaceusing a gaze interaction of claim 19, wherein determining that the gazepoint of the user has moved to a non-gaze input device comprises:determining that the gaze point of the user has moved to below the userinterface or a display on which the user interface is presented.